Modular prescription augmented reality display

ABSTRACT

In an embodiment, a modular augmented reality display is provided that incorporates prescription eyewear that can be used separately by the wearer. In an embodiment, an image is generated from a removable display attached to the eyewear and directed into the edge of a prescription lens, which acts as a waveguide. The image is internally reflected within the prescription lens, and is directed to the wearer by an image combiner embedded within the prescription lens. In an embodiment, the augmented reality display includes a wearable belt pouch that includes a battery and support electronics connected to the eyewear so that the weight on the eyewear is reduced.

BACKGROUND

Augmented reality (“AR”) is an emerging field in which graphicalelements are added to an image of the real world. Augmented realitydisplays can be constructed in a number of ways. For example, somehandheld devices implement a type of augmented reality by capturing animage of the real world with a camera, adding computer-generated imagesto the captured image, and then displaying the augmented image on themobile display. Other devices attempt to add a generated image toeyeglasses similar to a heads-up display. Creating effective AR displaysis particularly difficult when the user of the display uses a form ofcorrective vision. Allowing for use with existing corrective eyewearoften results in a bulky and cumbersome AR display, and using the ARdisplay without corrective eyewear results in a blurry or uncleardisplay being presented to the wearer. Since a significant portion ofthe population relies on prescription eyewear to see properly, producingan effective AR display that allows for vision correction is animportant problem.

BRIEF DESCRIPTION OF THE DRAWINGS

Various techniques will be described with reference to the drawings, inwhich:

FIG. 1 illustrates an example of a prescription augmented realitydisplay worn by a user, in accordance with an embodiment;

FIG. 2 illustrates an example of an augmented reality display integratedinto prescription eyeglasses, in accordance with an embodiment;

FIG. 3 illustrates an example of an image produced by an augmentedreality display, in accordance with an embodiment;

FIG. 4 illustrates an example of a lens assembly of an augmented realitydisplay, in accordance with an embodiment;

FIG. 5 illustrates an example of a display unit and lens geometry thatcombines the output of the display unit with a prescription lens to forman augmented reality display, in accordance with an embodiment;

FIG. 6 illustrates an example of a prescription lens assembly with animage combiner, in accordance with an embodiment;

FIG. 7 illustrates an example of prescription augmented reality for amyopic eye, in accordance with an embodiment;

FIG. 8 illustrates an example of a spot diagram for a prescription lenswith and without CYL power, in accordance with an embodiment;

FIG. 9 illustrates an example of the angular resolution of a foveated ARdisplay, in accordance with an embodiment;

FIG. 10 illustrates an example of a process that, as a result of beingperformed, optimizes a prescription lens of a prescription AR display,in accordance with an embodiment;

FIG. 11 illustrates an example of a process that, as a result of beingperformed, optimizes an AR display of a prescription AR display, inaccordance with an embodiment;

FIG. 12 illustrates an example of a design trade-off space for lensthickness and eye box size for a prescription AR display, in accordancewith an embodiment;

FIG. 13 illustrates an example of a correlation between a display panelshift and an image plane shift for a prescription AR display, inaccordance with an embodiment;

FIG. 14 illustrates an example of a design trade-off space for avertical field of view and eye relief for a prescription AR display, inaccordance with an embodiment;

FIG. 15 illustrates an example of an AR display with an adjustable imageposition, in accordance with an embodiment;

FIG. 16 illustrates an example of the angular resolution of a foveateddisplay, in accordance with an embodiment;

FIG. 17 illustrates an example of a modular augmented reality displaywith an upward configuration that allows the separate use of aprescription eyeglass component, in accordance with an embodiment;

FIG. 18 illustrates an example of a modular augmented reality displaywith an outward configuration that allows the separate use of aprescription eyeglass component, in accordance with an embodiment;

FIG. 19 illustrates an example of a detachable display assembly of anaugmented reality display, in accordance with an embodiment;

FIG. 20 illustrates an example of a modular augmented reality displaybeing worn by a user, in accordance with an embodiment;

FIG. 21 illustrates an example of a modular augmented reality displaythat communicates with a computer server, in accordance with anembodiment;

FIG. 22 illustrates an example of a parallel processing unit (“PPU”), inaccordance with an embodiment;

FIG. 23 illustrates an example of a general processing cluster (“GPC”),in accordance with one embodiment;

FIG. 24 illustrates an example of a memory partition unit, in accordancewith one embodiment;

FIG. 25 illustrates an example of a streaming multi-processor, inaccordance with one embodiment; and

FIG. 26 illustrates a computer system in which the various examples canbe implemented, in accordance with one embodiment.

DETAILED DESCRIPTION

The present document describes a modular augmented reality (“AR”)display that includes a prescription lens for vision correction. In atleast one embodiment, the prescription lens is integrated into a frame,and the lens-frame assembly can be detached from AR components of the ARdisplay so that the lens-frame assembly can be used as a set ofconventional prescription eyewear without AR functionality. Removing theAR components lightens the eyewear sufficiently so that the eyewear ismore comfortable in extended use. In various examples, the electroniccomponents are removably attached using magnets or clips to thelens-frame assembly. In at least one embodiment, the removable ARcomponents include one or more organic light emitting diode (“OLED”)displays and one or more lenses that shape and direct images into theedge of the prescription lens, thereby providing AR image content to thewearer.

In some examples, the AR components include a support pack with abattery, computer system, and wireless interface. In at least oneembodiment, a server computer system sends display information to thecomputer system on the support pack via the wireless interface, and thecomputer system sends the information to the AR components mounted onthe eyewear. In various embodiments, the support pack can be attached toa belt, pant waistline, or article of clothing worn by the user. In someexamples, the support pack may be placed in a user's pocket. In at leastone embodiment, the support pack is connected to AR components on thelens-frame assembly via a wired connection that carries power and datasignals. By moving the battery and computer components to the supportpack, the weight of the AR display on the wearer's head is reduced,increasing comfort in long-term use.

In at least one embodiment, the AR display presents virtual images inreal-world scenes while preserving the viewer's natural vision. In atleast one embodiment, an optical structure is provided that has a slimform factor, a high-resolution, large field-of-view (FOV), large eyebox, and variable focus. In an embodiment, the diverse spectrum of humanhead shape and eye structure aggravates this challenge further. Varioususers have different interpupillary distance (IPD, 54-68 mm) and noseshapes, which raise the bar on eye box and eye relief coverage beyondthe requirement for a single user. More than 40% of the population usesspecial aids for vision correction caused by myopia, hyperopia,astigmatism, and presbyopia. Unlike other designs that may be used withprescription eyeglasses, at least one embodiment described hereinprovides an AR display that includes corrective lenses adapted to theviewer's prescription. By integrating the viewer's prescription into theAR display, overall weight and size of the system can be improvedsignificantly.

In an embodiment, a prescription-embedded AR display is provided. In anembodiment, an optical design for the AR display utilizes a prescriptionlens as a waveguide for the AR display. In an embodiment, a free-formimage combiner is embedded in the prescription lens, allowing theone-piece lens to both deliver virtual scenes and also correct thevision of a real-world scene simultaneously. In at least one embodiment,the image combiner is a half-silvered mirror or semi-reflective film. Inan example of an embodiment, a shape for the prescription lens isprovided for a modified myopia eye model. In an embodiment, a free-formimage combiner, in-coupling prism, and beam-shaping lens are optimizedbased on each individual prescription lens. In at least one embodiment,a customized ergonomic eye-glasses design is achieved by using 3D facialscanning. In at least one embodiment, a Prescription AR prototype with a5-mm thick lens provides 1 diopter (1D) vision correction, 23 cycles perdegree (cpd) angular resolution at center, 4 mm eye box, and varifocal(0D-2D) capability. In at least one embodiment, the prototype islightweight (169 g for dynamic and 79 g for static prototype), has 70%transparency, protects a user's privacy, and enables eye-contactinteraction with surroundings.

FIG. 1 illustrates an example of a prescription augmented realitydisplay worn by a user, in accordance with an embodiment. In at leastone embodiment, the wearer 102 equips the augmented reality display in amanner similar to conventional eyeglasses. In at least one embodiment,at least one augmented reality lens 104 is held by a frame 106 which isworn by the wearer 102. In at least one embodiment, two augmentedreality lenses are provided in the augmented reality display to providebinocular vision of the augmented images.

In at least one embodiment, the augmented reality display positions thelenses close to the wearer's face so that the eye relief and eye box ofthe display can be minimized. In at least one embodiment, the augmentedreality lens 104 provides three regions through which the wearer canlook. In at least one embodiment, a first region 108 at the top of theaugmented reality lens and a third region 112 at the bottom of theaugmented reality lens provide an optical correction in accordance witha corrective vision prescription of the wearer. In at least oneembodiment, a second region 110 in the middle of the augmented realitylens includes an image combiner constructed from a half mirrored surfaceembedded within the lens. In at least one embodiment, an image injecteddown through the edge of the lens from a display within the frame isreflected internally within the lens until it comes in contact with theimage combiner. In at least one embodiment, the image is redirectedtowards the eye of the wearer along with an image of the real world thatis transmitted through the lens and the image combiner. In at least oneembodiment, the first region 108 and the third region 112 are coatedwith a neutral density filter so that the image transmission through thelens is roughly even from the top to the bottom of the lens. In at leastone embodiment, the view through the second region 110 includes bothtransmitted images from the real-world and images generated from theelectronic display within the frame of the augmented reality display.

In at least one embodiment, the profile of the augmented reality lens104 is adapted in accordance with a vision prescription of the wearer.In at least one embodiment, the image combiner is formed within the lensand has a surface profile that similarly presents an in-focus image inaccordance with the vision correction needed by the wearer. In at leastone embodiment, the position of the virtual image originating fromwithin the display on the augmented reality display can be altered bymoving the position of the display in relation to the frame eithermanually or by electronic servo.

FIG. 2 illustrates an example of an augmented reality display integratedinto prescription eyeglasses, in accordance with an embodiment. In atleast one embodiment, the augmented reality display is produced in aneyeglass form factor having a frame with two arms 202 and 204. In atleast one embodiment, a power and control unit 206 drives a pair ofelectronic displays mounted in the top of the frame. In at least oneembodiment, the power and control unit 206 includes one or more buttonsto allow control of the display by the wearer. In at least oneembodiment, the power and control unit 206 includes a battery,processor, and graphical interface. In at least one embodiment, theprocessor and graphical interface may include a CPU or GPU as describedin FIGS. 16 through 20. In at least one embodiment, the augmentedreality display includes two lenses 208 and 210 that include an imagecombiner within each lens. In at least one embodiment, the imagecombiner combines the image transmitted through the lens with the imagecreated by the electronic display. In at least one embodiment, theaugmented reality display includes a beam-shaping lens and prism, 212and 214, in each lens that directs the image produced by the electronicdisplay downward into the lens such that the image is internallyreflected and directed to the image combiner of each lens.

FIG. 3 illustrates an example of an image produced by an augmentedreality display, in accordance with an embodiment. In at least oneembodiment, a first image 302 illustrates an image transmitted throughthe lens from the real-world to the wearer. In at least one embodiment,a second image 304 illustrates an augmented image seen by the wearer ofthe augmented reality display. In the example illustrated in FIG. 3, thesecond image includes a logo and lettering added by the image combinerand produced by an electronic display 306 on the augmented realitydisplay.

FIG. 4 illustrates an example of a lens assembly of an augmented realitydisplay, in accordance with an embodiment. In at least one embodiment,the augmented reality display includes a free-form image combiner 402embedded within a prescription lens 404. In at least one embodiment, theprescription lens 404 is fabricated in accordance with a visioncorrection prescription. In at least one embodiment, the free-form imagecombiner 402 is made by coating the interface between two parts of theprescription lens 404 with a half-silver material and joining the partswith optical glue.

In at least one embodiment, the lens assembly includes an in-couplingprism 406 and a beam-shaping lens 408 that, in combination, direct animage produced by a micro LED 412 into the edge of the prescription lens404. In at least one embodiment, the beam-shaping lens 408 is retainedin position by a lens holder 410 secured to a frame of the augmentedreality display. In at least one embodiment, the micro-LED is held inpace with a panel holder 414. In at least one embodiment, the panelholder 414 includes a manual or electronic actuator that allows themicro-LED to be moved along the optical axis of the beam-shaping lens.In at least one embodiment, movement of the micro-LED panel isaccomplished with a thumbscrew attached to the threaded rod. In at leastone embodiment, movement of the micro LED panel is accomplished using anultra-thin auto focus actuator module (“UTAF”).

FIG. 5 illustrates an example of a display unit and lens geometry thatcombines the output of the display unit with a prescription lens to forman augmented reality display, in accordance with an embodiment. FIG. 5illustrates a side view and the beam path of an embodiment. In at leastone embodiment, a prescription lens 502 works both for vision-correctionand as a wave-guide for an AR image produced by a micro display 504. Inat least one embodiment, light rays from the micro display 504 arerefracted by a beam-shaping lens 506 and enter the prescription lensthrough an in-coupling prism 508 to create a magnified virtual imagelocated a distance d_(i) from an eye 510. In at least one embodiment,the virtual image depth can be dynamically changed from 0D to 2D bymoving the micro display 504 axially (Δa).

In at least one embodiment, a prescription AR display optically correctsa user's vision with a prescription lens, and utilizes the prescriptionlens as a waveguide in an AR display system. As shown in FIG. 5, the topsurface of the prescription lens of thickness t₁ is used as an entranceof the waveguide. In at least one embodiment, the light rays from amicro display of size w_(d)×h_(d) and resolution N_(x)×N_(y) located infront of the user's forehead with an angle θ_(d) are refracted by abi-convex (R_(l1), R_(l2)) beam-shaping lens (refractive index: n₁,thickness: t_(BSL)) located at a distance a from the micro display withthe tilted angle θ_(l) and entered to the waveguide through anin-coupling prism (refractive index: n₁) located at d_(p) from thebeam-shaping lens with the tilted angle θ_(p). In at least oneembodiment, the in-coupling prism is composed by a set of aplano-concave and a convex-plano cylindrical lens. In at least oneembodiment, the rays are refracted by a cylindrical lens (R_(cy),refractive index: n₂) located at t_(c) from the prism surface with thetilted angle θ_(c) and travel in the waveguide (refractive index: n₃,tilted angle: θ_(w)) located at t_(w) from the cylindrical surface asshown in FIG. 4. In an embodiment, the light rays are internallyreflected twice by the frontal surface (S_(f)) and the rear surface(S_(r)) of the prescription lens, reflected by a free-form half-mirrorcoated surface (S_(free), tilted angle: θ_(f)), and arrive at the pupilof the eye. In at least one embodiment, the in-coupling prism,cylindrical lens, upper part of the prescription lens, and lower part ofthe prescription lens are bonded by an optical adhesive, such that theprescription lens is comprised of two lens pieces: the main lens and thebeam-shaping lens. In at least one embodiment, the enlarged virtualimage of size w_(i)×h_(i) is located at distance d_(i) from the eye inthe vision-corrected real scene. In at least one embodiment, the virtualimage depth can be dynamically adjusted (Δd_(i)) by moving the microdisplay back and forth (Δa).

In at least one embodiment, the optical design process includes twophases: the prescription lens design (S_(f), S_(r)) and the AR displaypath design using an optics simulation tool such as Zemax OpticStudio.The overall optical path is difficult to investigate using an analyticform because of the free-form surface and the multiple off-axiscomponents utilized in the display. Nevertheless, in at least oneembodiment, a universal design and optimization method is demonstratedwhich is valid for many prescriptions including myopia, astigmatism,hyperopia, and presbyopia. FIGS. 10-11 show an example of the two-phaseoptimization process in Prescription AR, which is started from theuser's eyeglasses prescription including spherical correction (SPH),cylinder correction (CYL), axis of astigmatism (AXIS), and add power(ADD).

FIG. 6 illustrates an example of a prescription lens assembly with animage combiner, in accordance with an embodiment. In at least oneembodiment, the prescription lens is constructed using two injectionmolded pieces. In at least one embodiment, a first lens part 602 and asecond lens part 604 are joined with optical glue at an interface. In atleast one embodiment, a half-silvered image combiner is placed withinthe interface and is embedded within the assembled lens.

FIG. 7 illustrates an example of prescription augmented reality for amyopic eye, in accordance with an embodiment. In at least oneembodiment, the first phase is the optimization of the prescriptionlens, including the frontal (S_(f)) and rear (S_(r)) surface profile.FIG. 7 shows, in one embodiment, how to design the prescription lens fora myopic eye. As shown in FIG. 7, the normal vision whose amplitude ofaccommodation is 4D has a far point at 0D and near point at 4D. The 1Dmyopic eye with the same amplitude of accommodation has a far point at1D and near point at 5D. So the observer cannot perceive full-resolutionimages of the objects located at 0.6D. In at least one embodiment, theprescription lens shifts the object at infinity to the myopic eye's farpoint (1D) so that the objects are imaged at 1.6D plane, inside thedepth of field (DOF). In at least one embodiment, the prescription lenscompensates for astigmatism by adding inverse cylinder power to thegiven axis.

In at least one embodiment, instead of using direct calculation ofsurface profiles from the SPH, CYL, AXIS, and ADD values, both surfacesare determined using a human eye model. In at least one embodiment, thisdetermination method reduces the aberration at the given thicknesst_(l), refractive index n₃, and given eye relief d_(e). Atchison built ahuman myopic eye model based on the measured data from 121 subjects, andit is known in the art that the total astigmatism is the sum of thecorneal and internal astigmatism. However, there isn't a general humaneye model covering both myopia and astigmatism. The techniques presentedin the present document assume corneal astigmatism only and modified thecorneal surface property of the Atchison's model. This assumption isvalid in this case because the prescription lens is affected by the sumof the astigmatism, not the source. The cornea surface profile C_(rv)and C_(rh) are calculated from the CYL and AXIS value, and the modifiedeye model is achieved with SPH value as shown in Table 1 below.

In at least one embodiment, Table 1 below shows the modified myopia eyemodel based on Atchison's model where the r_(x) and r_(y) are the radiusvalues of a bifocal system in horizontal and vertical respectively,k_(x) and k_(y) are the conic constant of bifocal system in horizontaland vertical respectively, N_(d) is the reflective index of material,and V_(d) is the Abbe number of the material. In at least oneembodiment, the radius of cornea surface, r*_(x), is calculated byadding the CYL power into another direction as D_(x)=D_(y)+CYL, wherethe D_(y)=(Nd−1)/r_(y). The compete equation of r_(x) is expressed belowas Eq(1).

Surface Type Radius Conic Thickness Material Rotation Cornea biconicr*_(x) = (Nd − 1)/D_(x) k_(x) = −0.15 0.55 Nd = 1.376 90-AXIS (^(⋄))r_(y) = 7.77 + 0.022SR k_(y) = −0.15 Vd = 55.468 Aqueous standard  6.40 −0.275 3.05 Nd = 1.3337 — Vd = 50.522 Stop standard infinite — 0.1 Nd =1.337 — Vd = 50.522 Anterior gradient 11.48 −5.00 1.44 1.371 +0.0652778Z — lens lens −0.0226659Z² −0.0020399(X² + Y²) Posteriorgradient infinite — 2.16 1.418 − 0.0100737Z² — lens lens −0.0020399(X² +Y²) Vitrous standard −5.90 −2.00 16.28 − 0.299SR Nd = 1.336 — Vd =51.293 Retina biconic r_(x) = −12.91 − 0.094SR k_(x) = 0.7 + 0.026SR — —— r_(y) = −12.72 + 0.004SR k_(y) = 0.225 + 0.017SR

In at least one embodiment, based at least in part on this modifiedmyopia eye model, S_(f) and S_(r) are determined. In at least oneembodiment, S_(f) is set as a spherical surface of radius r_(f) whileS_(r) is set as a bifocal surface of radii r_(ro), r_(re) and rotationangle θ_(r), to correct the myopia and the astigmatism. In anembodiment, the values were optimized iteratively with the meritfunction for the range of 12 to 20 mm eye relief and 26×18 degrees ofthe field.

In at least one embodiment, FIG. 8 shows a spot diagram change aroundthe retina plane of a myopic astigmatism eye (SPH: −2, CYL: −2, andAXIS: 30) without and with the prescription lens. In at least oneembodiment, compared to the naked eye and the myopia-only correctionlens, the designed prescription lens forms a smaller focal point at theretinal plane as shown in FIG. 8. In at least one embodiment, FIG. 8illustrates a spot diagram on the retina through focus shifting: (a) theblurred spot on the retina from the infinite object without prescriptionlens, (b) the focused spot on the retina only optimized for SPH, and (c)optimized for SPH, CYL, and AXIS.

$\begin{matrix}{r_{x}^{*} = \frac{r_{y} \times \left( {{Nd} - 1} \right)}{\left( {{Nd} - 1} \right) + {{CYL} \times r_{y}}}} & (1)\end{matrix}$

In at least one embodiment, based at least in part on the prescriptionlens design, other geometric parameters (R_(l1), R_(l2), R_(cy), a,d_(p), t_(BSL), t_(c), t_(w), θ_(d), θ_(l), θ_(p), θ_(c), θ_(w),θ_(free), and S_(free)) are optimized in the second phase. In at leastone embodiment, although actual numbers will be calculated by a toolsuch as Zemax OpticStudio, the geometry of optics, the materials, theconstraints, and the priority (merit function) should be carefullyconsidered at the design stage for the best performance.

In at least one embodiment, FIG. 5 shows a detailed diagram of the ARdisplay path. In at least one embodiment, in the waveguide, the lightrays are reflected at the positive power surface (S_(f)) first, and atthe negative power surface (S_(r)) second. In at least one embodiment,it is reasonable to choose a positive power image combiner (S_(free))for the freeform surface to produce a flatter focal plane, symmetricpower distribution, and less aberration. In at least one embodiment, thefreeform surface is characterized by an extended polynomial equationincluding conic aspherical surfaces and extended polynomial terms asfollows:

$\begin{matrix}{{z = {\frac{{cr}^{2}}{1 + \sqrt{1 - \left( {\left( {1 + k} \right)c^{2}r^{2}} \right)}} + {\sum\limits_{i}^{N}{A_{i}{E_{i}\left( {x,y} \right)}}}}},} & (2)\end{matrix}$

where c is the curvature for the base sphere, r is the normal radiusexpressed as r=√{square root over (x²)}+y², k is the conic constant, Nis the number of polynomial terms, and A_(i) is the coefficient of thei^(th) extended polynomial terms as Eq(3). In at least one embodiment,as part of the optimization in freeform surface, the 4^(th) polynomialhas been considered, in which N=16 in the Eq(3) below.

Σ_(i=0) ^(N) A _(i) E _(i)(x,y)=A ₀ +A ₁ x ¹ y ⁰ +A ₁ x ⁰ y ¹ +A ₃ x ² y⁰ +A ₄ x ₁ y ₁ +A ₅ x ⁰ y ²+ . . .   (3)

In at least one embodiment, the bi-convex beam-shaping lens increasesthe system's numerical aperture (NA) for higher resolution andcompactness (shorter optical path). In at least one embodiment, thein-coupling prism guides the light rays into the waveguide with thetotal internal reflection condition. In at least one embodiment, they-axis only cylindrical surface (R_(cy)) inside the in-coupling prismcompensates the astigmatism and the tilted image plane, which are causedby the off-axis folded path. In at least one embodiment, the tiltedangle of the beam-shaping lens is identical to the tilted angle of themicro-display for the symmetric magnification (θ_(d)=θ_(t)), but theangles of other components were freely decided by the optimizer tomaximize FOV and minimize aberration. In at least one embodiment, thematerials for the beam shaping lens and the upper part of thein-coupling prism (n₁, v₁), the lower part of that (n₂, v₂), and theprescription lens (n₃, v₃), where n and v refer to index of refractionand Abbe number respectively, were carefully chosen to minimize thethicknesses and the chromatic aberration using the different dispersioncharacteristics. In at least one embodiment, the distances (a, d_(p),t_(BSL), t_(c), t_(w)) were calculated to some non-negative values basedon various constraints and the priorities.

In at least one embodiment, the optical configuration for AR function islimited by giving the constraints for the optical system in the Meritfunction. In at least one embodiment, the constraints are determined bythe comprehensive consideration of lens implementation, distance fromforehead, total internal reflection (TIR) inside prescription lens, andboundary on display panel. In at least one embodiment, the centerthickness and edge thickness of each lens, t_(BSL), t_(c), t_(w), arelimited to more than one mm for the manufacturability of the lens. In atleast one embodiment, constraints for the air thickness, a, dp, arelimited to more than 0.2 mm to avoid the superposition of the lens. Inat least one embodiment, the sum of thickness a, d_(p), t_(BSL), t_(c),and t_(w) are limited to within 8.5 mm to minimize total thickness of ARsystem.

FIGS. 10-11 illustrate an embodiment of the two-phase optimization of aprescription AR display. In at least one embodiment, the frontal andrear surfaces (S_(f), S_(r)) are optimized at given lens thickness t_(l)in the first step based on user's prescription. In at least oneembodiment, other geometric parameters (R_(l1), R_(l2), R_(cy), a,d_(p), t_(BSL), tc, t_(w), θ_(d), θ_(l), θ_(p), θ_(c), θ_(w), andθ_(free)) are optimized in the second step based on target foveatedresolution and eye relief range d_(e).

FIG. 10 illustrates an example of a process 1000 that, as a result ofbeing performed, optimizes a prescription lens of a prescription ARdisplay, in accordance with an embodiment. In at least one embodiment,at block 1002, a prescription is obtained for a wearer. The prescriptionmay be, in various embodiments, a prescription for myopia, astigmatism,presbyopia, or various combinations of vision problems. In at least oneembodiment, at block 1004, a mathematical model for the eye is generatedin accordance with the obtained prescription. In the present document,an example for a myopic eye is presented. In at least one embodiment, atblock 1006, a script for Zemax Studio macro as described below isexecuted with appropriate vision correction parameters. In at least oneembodiment, at block 1008, the bifocal power and rotation angle of thez-axis in the cornea surface are determined. In at least one embodiment,at block 1010, the prescription lens surface is initialized, and atblock 1012, the Merit function criterion is set. In at least oneembodiment, at block 1014, the bifocal radius and rotated angle of theprescription lens surface is optimized, and at block 1016, optimizationof the prescription portion of the AR display is complete.

FIG. 11 illustrates an example of a process 1100 that, as a result ofbeing performed, optimizes an AR display of a prescription AR display,in accordance with an embodiment. In at least one embodiment, at block1102, the prescription lens parameters determined above are obtained. Inat least one embodiment, at block 1104, the relationship between theimage combiner and the display panel is modeled. In at least oneembodiment, the relationship establishes a foveated image that ispresented to the wearer. In at least one embodiment, at block 1106, theparameters defining the AR display, including the image combiner, aredetermined. In at least one embodiment, at block 1108, the constant ofthe merit function is established. In at least one embodiment, at block1110, the resolution distribution of the foveated display is set. In atleast one embodiment, the resolution of the foveated display is achievedas a result of the electronic micro-display being flat, and the opticsof the beam shaping lens and optical path correcting the shape of thedisplay to match the image combiner. In at least one embodiment, theresolution density presented to the wearer is illustrated in FIG. 9. Inat least one embodiment, at block 1112, the polynomial coefficientsurface performance of the surface is optimized. In at least oneembodiment, at block 1114, due to the adjustment of the foveateddisplay, a center resolution of over 23 cycles per degree (“CPR”) isachieved. In at least one embodiment, at block 1116, the AR displayoptimization is complete.

In at least one embodiment, where the thickness of the prescription lensis 5 mm, the size of the free-form combiner is limited, especially inthe vertical field of view. In at least one embodiment, the thickerprescription lens allows a larger field of view by the larger size ofthe combiner. In at least one embodiment, although it is complicated toevaluate the FoV from the free-form surface and reflection constraint ofthe light path, tools such as Zemax Studio provide an effective way toget the vertical field of view. As shown in FIGS. 12-14, the trend lineis linear to the thickness of the prescription lens. In at least oneembodiment, the eye box size grows with lens thickness.

FIGS. 12-14 illustrate an embodiment of a design trade-off space for theprescription AR display. In at least one embodiment, the micro displayw_(d)×h_(d)=10.08×7.56 mm and pixel pitch 6.3 μm, virtual image planed_(i)=1D, and thickness t_(l)=5 mm. FIG. 12 illustrates thickness vs.FOV and eye box. In at least one embodiment, both FOV and eye box areproportional to the t_(l). FIG. 13 illustrates focus cue. In at leastone embodiment, the virtual image plane can be changed back and forth(Δd) with the axial movement of the micro display (Δa). FIG. 14illustrates Eye Relief vs. FOV. In at least one embodiment, smaller eyerelief provides larger FOV

FIG. 9 illustrates an example of the angular resolution of a foveated ARdisplay as viewed by the wearer, in accordance with an embodiment. In atleast one embodiment, the AR display is a fixed foveated display. In atleast one embodiment, the AR display is adjusted to the foveatedresolution distribution, which presents a high angular resolution, 26CPR, at the foveal region and a low angular resolution, 3 cpd, outsideof foveated region in optics. In at least one embodiment, the panel usedis a 6.3 μm micro-OLED, which can reach to 23 CPR of angular resolution.In at least one embodiment, an optimization is performed done in ZemaxStudio by giving the foveated weight of optimized priority order for thefield. In at least one embodiment, the special frequency date of MTF30is extracted from Zemax Studio and the data is converted into angularresolution by calculating the field of view of the AR image, as shown inFIG. 9. In at least one embodiment, the high-resolution region is notsymmetric because it is hard to perfectly compensate the AXIS angle ofastigmatism. FIG. 9 illustrates foveated optimization of angularresolution over FOV for a 1D myopia prescription AR display.

In at least one embodiment, using the techniques described herein, aperson of ordinary skill in the art is able to adjust the prescriptionAR display to correct for most vision problems including myopia,astigmatism, presbyopia, and various combinations these problems. Invarious embodiments, the prescription AR is adaptable to multiple myopiacases (0D (normal vision), 1D, 2D, 3D, 4D, and 5D) and a myopicastigmatism case (SPH=2D, CYL=2, AXIS=30). Table 2 shows designparameters for one embodiment of a 1D myopia Prescription AR display.Table 2 shows the geometric and optical parameters of one embodiment ofprescription AR for 1D myopia.

TABLE 2 The designed performance of implemented prescription embedded ARdisplay. Items Units Values Prescription diopther −1 Lens thickness mm 5Eye relief mm 20 Eye box mm 4 Image object m, through the lens 1 Fieldof view degrees 20 × 40 Resolution cycles per degree(CPR) 23 at center

In at least one embodiment, the fabrication of the optical componentsmay be accomplished with the following techniques. In at least oneembodiment, since facial structure is unique to the wearer, theergonomic frame design is as important as the optics design. In at leastone embodiment, the optics for the eye relief of the AR display isoptimized in the range of 12 mm to 20 mm. In at least one embodiment,however, smaller eye relief can provide a larger FOV and a morecomfortable fit (closer center of mass). In at least one embodiment, thecenter of the pupil should be aligned with the optical axis for asuperior foveated experience. In at least one embodiment, frame designof the glasses should be chosen in accordance with the wearer'sinterpupillary distance (“IPD”) too.

In at least one embodiment, the facial structure of the intended weareras illustrated in FIG. 1 is 3D-scanned with a 3D-Camera such as theKinect sensor from Microsoft, and imported to 3D-rendering software(such as Fusion 360), and the glasses frame is designed and optimizedfor each user. In at least one embodiment, the glasses frame designs areparameterized with the input of the IPD and the width of the head,followed by fitting if the nose pieces.

In at least one embodiment, an OLED-based dynamic prototype was createdwith the following features. In at least one embodiment, two 10.08×7.56mm Sony micro OLED (ECX339A) displays were used as binocular microdisplays, where each display has 1600×1200 resolution, 6.3 μm pixelpitch, and maximum brightness 1000 cd/m2. In at least one embodiment,the free-form optics with the 70% transparency for 1D myopia werefabricated. In at least one embodiment, a 3D-printed frame housed andaligned the optical structures including main lens+in-coupling prism,beam-shaping lens, micro display, and driving board. In at least oneembodiment, a 3D printed gear was also applied to change the IPD. In atleast one embodiment, the weight of the dynamic prototype including thedriving board was 164 g.

In at least one embodiment, an LUT-based static prototype was createdwith the following features. In at least one embodiment, two sets of a10.08×7.56 mm, 3048 pixel per inch light valve technology (LUT) filmwith an ElectroLuminescent (EL) film back light were used for the staticdisplay. In at least one embodiment, a CR-2032 coin cell powered both ELfilms. In at least one embodiment, a 3D-printed housing aligned all ofthe optics, statics display modules, and the battery for wearable eyeglasses form factor. In at least one embodiment, the weight of thestatic prototype was 79 g.

In at least one embodiment, the image content for the prototype is abinocular image. In at least one embodiment, the binocular image isproduced by a G3D Innovation Engine which is a powerful rendering enginewith the open source of C++ program. In at least one embodiment, therendering engine supports the image rendering of virtual reality thatallows the customer to add the scene by a virtual reality platform suchas VRapp. In at least one embodiment, in the coding of the virtualreality platform, the field of view, depth of focus, pupillary distance,and resolution are set with same parameter of the prescription-embeddedAR display. In at least one embodiment, the field of view measurement ofthe AR image covered 20 by 40 degrees in the vertical and horizontaldirection respectively.

In at least one embodiment, the AR display achieves corrected vision. Inan experiment conduced on a prototype of an embodiment, a scene fordifferent real objects including a car, a horse, and an eye chart with adistance of 0.5 m, 1 m, and 3 m respectively, was used. In at least oneembodiment, in order to imitate a wearer who has a 1 diopter myopia eye,the camera focused on the car in the scene without theprescription-embedded AR display. The clear details on the car show thata 1 diopter myopia eye is able to clearly view an object at 0.5 m. In atleast one embodiment, using the prescription-embedded AR display, thefocus point shifted to 1 m to target the horse, without changing thesetting of focus on the camera. The focus shift amount demonstrates thatthe prescription lens has −1 diopter power. In at least one embodiment,a clear AR image is presented by the display panel at 1 m distancethrough the prescription embedded AR display, and the eye chart lookssharper due to the contribution of vision correction.

FIG. 15 illustrates an example of an AR display with an adjustable imageposition, in accordance with an embodiment. In at least one embodiment,the AR display can be adapted to operate as a vari-focal AR display. Asshown in FIG. 15, a three-position Micro-OLED (“MOLED”) panel 1502 withsize, Pd, and shifting distance, δd, makes the different depth of ARimage at d1, d2, and d3 from the eye, where equal to 0.5 m, 1 m, and 3m, respectively. In at least one embodiment, three kinds of thickness ofassemblies for Micro-OLED panel provide the shifting amount of 0.3 mm,δd. In at least one embodiment, the depth of the clear AR image iscontrolled and produces a vari-focal display that reduces the problem ofvergence and accommodation conflict (VAC) in the near-eye displaysystem.

In at least one embodiment, a vision correction function is an importantproperty in a near-eye display system. In at least one embodiment, an ARdisplay is directly integrated into a prescription lens. In at least oneembodiment, each individual AR display is adapted to include acustomized prescription lens in accordance with a prescription of SPH,CYL, and AXIS. In at least one embodiment, the configuration of theoptical components can be used for the myopia patient with 0 to −7diopter, −2 diopter of astigmatism, and rotation angle of astigmatism.

In at least one embodiment, a free-form image combiner is made bymolding a prescription lens in two pieces and coating the interfacebetween the two pieces with anti-reflective coating and 30% of ND filtercoating. In at least one embodiment, the system achieves a field of viewof 20 by 40 degrees and a foveated resolution distribution of 23 CPR inthe foveal region. In at least one embodiment, the eye box size is 4 mm.In at least one embodiment, the prescription-embedded AR displaydescribed herein offers both corrected vision and a clear AR image at 1m. In at least one embodiment, the depth of the AR image is adjustablefrom 0.5 m to 3 m by applying a corresponding 0.3 mm shift to theposition of the display panel.

In at least one embodiment, the prescription embedded AR display is acompact design, which provides a volume of 6.5 cm3 for the opticalengine, including the 5 mm thickness prescription lens, other opticalelements, and a micro-OLED. In at least one embodiment, the prescriptionembedded AR display achieves vision correction for the environment sceneand also gives a clear AR image for the wearer.

FIG. 16 illustrates an example of the angular resolution of a foveateddisplay, in accordance with an embodiment. The graph illustrates anembodiment where the perceived resolution of AR content is higher towardthe center of the field of view and lower toward the edges of the fieldof view.

FIG. 17 illustrates an example of a modular augmented reality displaywith an upward configuration that allows the separate use a prescriptioneyeglass component, in accordance with an embodiment. In at least oneembodiment, the augmented reality display is produced in an eyeglassform factor having a frame with two arms 1702 and 1704. In at least oneembodiment, a power and control unit 1706 drives a pair of electronicdisplays 1716 and 1718 removably mounted to the top of the frame. In atleast one embodiment, the power and control unit 1706 includes one ormore buttons to allow control of the display by the wearer. In at leastone embodiment, the power and control unit 1706 is connected to a remotesupport pack worn by the user that includes a battery, microcomputer,and wireless interface. In at least one embodiment, the microcomputermay include a CPU or GPU as described in FIGS. 22 through 26. In atleast one embodiment, the augmented reality display includes two lenses1708 and 1710 that include an image combiner within each lens. In atleast one embodiment, the image combiner combines the image transmittedthrough the lens with the image created by the electronic display. In atleast one embodiment, the augmented reality display includes abeam-shaping lens and prism (1712 and 1714) in each lens that directsthe image produced by the electronic display downward into the lens suchthat the image is internally reflected and directed to the imagecombiner of each lens.

In at least one embodiment, the AR display includes connectors 1724 and1726 that allow the removal and attachment of the electronic displays1716 and 1718. In at least one embodiment, the connectors 1724 and 1726are magnetic attachments. In at least one embodiment, the magneticattachments include conductors that transmit electrical signals to andfrom the control unit and/or the support pack. In at least oneembodiment, the connectors are a clip, clasp, or slot that retain theelectronic displays 1716 and 1718. In at least one embodiment, theconnectors 1724 and 1726 include indexing features or keyways that aidin aligning the electronic displays 1716 and 1718 with the lenses 1708and 1710. In at least one embodiment, the eyeglasses include mechanicalalignment features to aid in attaching the display to the AR display tothe eyeglasses. In at least one embodiment, mechanical alignment isachieved using a mortise and tenon, one or more groves and ridges, orpins and matching holes. In at least one embodiment, the electronicdisplays 1716 and 1718 include an organic light emitting diode (“OLED”)display element coupled to one or more lenses and or prisms to directand shape an image.

In at least one embodiment, a pair of driving boards 1720 and 1722 areattached to the electronic displays 1716 and 1718. In at least oneembodiment, each driving board includes interface circuits and drivingcircuits for an OLED display. In at least one embodiment, the pair ofdriving boards 1720 and 1722 project upwards above the lenses. In atleast one embodiment, the driving boards connect to the electronicdisplays 1716 and 1718 via edge connectors.

In at least one embodiment, the control unit 1706 includes a battery,microcomputer, and interface for communicating with a server computersystem. In at least one embodiment, the interface is a wirelessBluetooth or WiFi interface. In at least one embodiment, the controlunit 1706 relays display information from the server computer system tothe pair of driving boards 1720 and 1722. In at least one embodiment,the control unit 1706 is removable in addition to other AR components toreduce the weight of the eyewear when AR features are not needed.

FIG. 18 illustrates an example of a modular augmented reality displaywith an outward configuration that allows the separate use ofprescription eyeglass component, in accordance with an embodiment. In atleast one embodiment, the augmented reality display is produced in aneyeglass form factor having a frame with two arms 1802 and 1804. In atleast one embodiment, a power and control unit 1806 drives a pair ofelectronic displays 1816 and 1818 removably mounted to the top of theframe. In at least one embodiment, the power and control unit 1806includes one or more buttons to allow control of the display by thewearer. In at least one embodiment, the power and c is connected to aremote support pack worn by the user that includes a battery,microcomputer, and wireless interface. In at least one embodiment, themicrocomputer may include a CPU or GPU as described in FIGS. 22 through26. In at least one embodiment, the augmented reality display includestwo lenses 1808 and 1810 that include an image combiner within eachlens. In at least one embodiment, the image combiner combines the imagetransmitted through the lens with the image created by the electronicdisplay. In at least one embodiment, the augmented reality displayincludes a beam-shaping lens and prism (1812 and 1814) in each lens thatdirects the image produced by the electronic display downward into thelens such that the image is internally reflected and directed to theimage combiner of each lens.

In at least one embodiment, the AR display includes connectors 1824 and1826 that allow the removal and attachment of the electronic displays1816 and 1818. In at least one embodiment, the connectors 1824 and 1826are magnetic attachments. In at least one embodiment, the magneticattachments include conductors that transmit electrical signals to andfrom the control unit and/or the support pack. In at least oneembodiment, the connectors are a clip, clasp, or slot that retain theelectronic displays 1816 and 1818. In at least one embodiment, theconnectors 1824 and 1826 include indexing features or keyways that aidin aligning the electronic displays 1816 and 1818 with the lenses 1808and 1810. In at least one embodiment, the electronic displays 1816 and1818 include an organic light emitting diode (“OLED”) display elementcoupled to one or more lenses and or prisms to direct and shape animage.

In at least one embodiment, a driving board 1820 is attached to theelectronic displays 1816 and 1818. In at least one embodiment, eachdriving board includes interface circuits and driving circuits for anOLED display. In at least one embodiment, the driving board 1820projects outwards from the lenses. In at least one embodiment, thedriving board 1820 connects to the electronic displays 1816 and 1818 viaedge connectors. In at least one embodiment, the driving board 1820 isconnected via a hinged connector allowing the board to pivot on an axisalong the top of the eyeglasses. By allowing the driving board 1820 topivot up and down, the position of the driving board 1820 can beadjusted to allow for the user wearing a hat with the AR display.

In at least one embodiment, the control unit 1806 is integrated into thesupport pack, and display signals are transmitted from the control packto the driving board 1820 and the electronic displays 1816 and 1818. Inat least one embodiment, the support pack includes user interfacebuttons that control the operation of the AR display.

FIG. 19 illustrates an example of a detachable display assembly 1916 ofan augmented reality display, in accordance with an embodiment. In atleast one embodiment, the augmented reality display includes a free-formimage combiner 1902 embedded within a prescription lens 1904. In atleast one embodiment, the prescription lens 1904 is fabricated inaccordance with a vision correction prescription. In at least oneembodiment, the free-form image combiner 1902 is made by coating theinterface between two parts of the prescription lens 1904 with ahalf-silver material and joining the parts with optical glue. In atleast one embodiment, the lens assembly includes an in-coupling prism1906.

In at least one embodiment, the detachable display assembly 1916includes a beam-shaping lens 1908 that directs an image produced by amicro LED 1912 into the edge of the prescription lens 1904. In at leastone embodiment, the beam-shaping lens 408 is retained in position by alens holder 1910 removably attached to a frame of the augmented realitydisplay. In at least one embodiment, the detachable display assembly1916 is attached to the prescription lens 1904 with a magneticattachment or mechanical clip. In at least one embodiment, the micro-LEDis held in pace with a panel holder 1914. In at least one embodiment,the panel holder 1914 includes a manual or electronic actuator thatallows the micro-LED to be moved along the optical axis of thebeam-shaping lens. In at least one embodiment, movement of the micro-LEDpanel is accomplished with a thumbscrew attached to the threaded rod. Inat least one embodiment, movement of the micro LED panel is accomplishedusing an ultra-thin auto focus actuator module (“UTAF”).

FIG. 20 illustrates an example of a modular augmented reality displaybeing worn by a user, in accordance with an embodiment. In at least oneembodiment, a user 2000 is wearing an AR display 2002 that integratedwith a set of prescription eyewear. In at least one embodiment, the user2000 wears a support pack 2004 which is connected to the AR display 2002via an electrical cable 2006. In at least one embodiment, the supportpack 2004 includes a battery, microcomputer, and wireless interfacecapable of communicating with a server that provides displayinformation. In at least one embodiment, the wireless interface is an802.11 or WiFi interface. In at least one embodiment, the wirelessinterface is a Bluetooth interface. In at least one embodiment, theelectrical cable 2006 is replaced or augmented by a Bluetooth ornear-field wireless connection between the support pack 2004 and the ARdisplay 2002.

In at least one embodiment, by moving various components to the supportpack 2004, the weight of the remaining components which are worn on theusers' head are approximately 50-100 grams. In at least one embodiment,the weight of the prescription eyewear only, without AR-relatedcomponents is less than 50 grams. It has been suggested that people thatwear prescription eyewear generally must do so for about 16 hours perday, and that in this extended use, a weight of more than 50 g becomesuncomfortable for many users. By moving heavy components such as thebattery to the support pack 2004, the AR headset can be comfortably usedfor many hours. If AR features are not needed, in some embodiments, thedisplay drivers and OLED displays may be detached from the eyewearreducing the weight of the eyewear even more.

FIG. 21 illustrates an example of a modular augmented reality displaythat communicates with a computer server, in accordance with anembodiment. In at least one embodiment, a user 2102 wears an AR displaythat includes a pair of arms 2106 and a prescription lens 2104, such asa prescription AR display described above. In at least one embodiment,the AR display includes a micro organic light emitting diode (“MOLED”)display 2108 and a display interface 2110 coupled to the display 2108.In at least one example, the display 2108 may be an LED, AMOLED, or LCDdisplay. In at least one embodiment, the display interface 2110 andMOLED display 2109 are removable by the user 2102, so that theprescription AR display can be used as a set of prescription eyewearwithout AR functionality. In at least one embodiment, the resultingeyewear is lightened, increasing comfort for the user 2102.

In various examples the display interface 2110 includes an electricalinterface that communicates with a support pack 2114 via a cable 2112.In at least one embodiment, the display interface includes electronicvideo display driver circuitry that converts signals obtained from thesupport pack 2114 into image data that is provided to the display 2108.In at least one embodiment, the display 2108 converts the image datainto images that are displayed to the user 2102 via the prescriptionlens 2104 as described above.

In at least one embodiment, the support pack 2114 includes a wirelessinterface 2116, a battery 2120, and a microcomputer 2118. In at leastone embodiment, the microcomputer 2118 may be a computer system havingone or more processors and memory containing instructions that, as aresult of being executed by the one or more processors, case themicrocomputer 2118 to acquire information from a computer server 2124,process the information, and send display information to the displayinterface 2110. In at least one embodiment, the battery 2120 is alithium ion, NiCad, Alkaline or other battery that provides power to thesupport pack. In some examples, the battery 2120 supplies power to thedisplay interface 2110 and display 2108 via the cable 2112. In someexamples, the cable 2112 supplies both serial data and power to thedisplay interface 2110 over a single wire of the cable 2112.

In at least one embodiment, the support pack 2114 communicates with thecomputer server 2124 using the wireless interface 2116. In at least oneembodiment, the wireless interface is a WiFi, cellular, or Bluetoothinterface. In at least one embodiment, the computer server 2124 providesdisplay information to the AR display via the support pack 2114. Thesupport pack lightens the weight on the user's 2102 head when AR imagesare being used. By providing a wireless connection between the supportpack 2114 and the computer server 2124, freedom of movement is providedto the user 2102 while using the AR display.

FIGS. 22-26 illustrate various systems that can be used to implementvarious embodiments of the invention. The systems illustrated anddiscussed in connection with FIGS. 22-26 may be used, for example, toexecute instructions to perform algorithms discussed herein includingbut not limited to image processing (e.g., object detection, objectrecognition, image segmentation, and other techniques) and calculatingdisplay information (e.g., the display to be rendered and/or a portionthereof, such as content to be added to an image captured through one ormore cameras). Software to implement the various techniques describedherein can be executed using one or more of the systems discussed belowand illustrated in connection with FIGS. 22-26.

In at least one embodiment, the AR display includes an electronicdisplay such as an organic light emitting diode (“OLED”), light emittingdiode (“LED”), light valve technology (“LUT”) display, or liquid crystaldisplay (“LCD”). In at least one embodiment, the electronic displayproduces an image which is directed through a beam-shaping lens andin-coupling prism into the edge of the lens of the AR display. In atleast one embodiment, the image is internally reflected within the lensby the surfaces of the lens until the image encounters a free-form imagecombiner located internally to the lens. In at least one embodiment, thesurface profile of the image combiner is constructed as described hereinso that both the image transmitted through the lens, and the imagegenerated by the AR display are presented to a wearer in accordance witha vision prescription for corrective eyewear.

In at least one embodiment, a computer system with one or moreprocessors is coupled to the electronic display, and the computer systemincludes memory and instructions that, when executed, cause the computersystem to generate electrical signals that are transmitted to theelectronic display. In at least one embodiment, the electrical signalsare converted by the electrical display into an image. In at least oneembodiment, an augmented reality graphics framework such as Spark AR,Wikitude, ARKit or ARCore on the computer system allows an applicationdeveloper to create software, that when run on the computer system,directs the addition of augmented reality elements on the AR display.

FIG. 22 illustrates a parallel processing unit (“PPU”) 2200, inaccordance with one embodiment. In an embodiment, the PPU 2200 isconfigured with machine-readable code that, if executed by the PPU,causes the PPU to perform some or all of the processes and techniquesdescribed throughout this disclosure. In an embodiment, the PPU 2200 isa multi-threaded processor that is implemented on one or more integratedcircuit devices and that utilizes multithreading as a latency-hidingtechnique designed to process computer-readable instructions (alsoreferred to as machine-readable instructions or simply instructions) onmultiple threads in parallel. In an embodiment, a thread refers to athread of execution and is an instantiation of a set of instructionsconfigured to be executed by the PPU 2200. In an embodiment, the PPU2200 is a graphics processing unit (“GPU”) configured to implement agraphics rendering pipeline for processing three-dimensional (“3D”)graphics data in order to generate two-dimensional (“2D”) image data fordisplay on a display device such as a liquid crystal display (LCD)device. In an embodiment, the PPU 2200 is utilized to performcomputations such as linear algebra operations and machine-learningoperations. FIG. 22 illustrates an example parallel processor forillustrative purposes only and should be construed as a non-limitingexample of processor architectures contemplated within the scope of thisdisclosure and that any suitable processor may be employed to supplementand/or substitute for the same.

In an embodiment, one or more PPUs are configured to accelerate HighPerformance Computing (“HPC”), data center, and machine learningapplications. In an embodiment, the PPU 2200 is configured to acceleratedeep learning systems and applications including the followingnon-limiting examples: autonomous vehicle platforms, deep learning,high-accuracy speech, image, text recognition systems, intelligent videoanalytics, molecular simulations, drug discovery, disease diagnosis,weather forecasting, big data analytics, astronomy, molecular dynamicssimulation, financial modeling, robotics, factory automation, real-timelanguage translation, online search optimizations, and personalized userrecommendations, and more.

In an embodiment, the PPU 2200 includes an Input/Output (“I/O”) unit2206, a front-end unit 2210, a scheduler unit 2212, a work distributionunit 2214, a hub 2216, a crossbar (“Xbar”) 2220, one or more generalprocessing clusters (“GPCs”) 2218, and one or more partition units 2222.In an embodiment, the PPU 2200 is connected to a host processor or otherPPUs 2200 via one or more high-speed GPU interconnects 2208. In anembodiment, the PPU 2200 is connected to a host processor or otherperipheral devices via an interconnect 2202. In an embodiment, the PPU2200 is connected to a local memory comprising one or more memorydevices 2204. In an embodiment, the local memory comprises one or moredynamic random access memory (“DRAM”) devices. In an embodiment, the oneor more DRAM devices are configured and/or configurable ashigh-bandwidth memory (“HBM”) subsystems, with multiple DRAM diesstacked within each device.

The high-speed GPU interconnect 2208 may refer to a wire-basedmulti-lane communications link that is used by systems to scale andinclude one or more PPUs 2200 combined with one or more CPUs, supportscache coherence between the PPUs 2200 and CPUs, and CPU mastering. In anembodiment, data and/or commands are transmitted by the high-speed GPUinterconnect 2208 through the hub 2216 to/from other units of the PPU2200 such as one or more copy engines, video encoders, video decoders,power management units, and other components which may not be explicitlyillustrated in FIG. 22.

In an embodiment, the I/O unit 2206 is configured to transmit andreceive communications (e.g., commands, data) from a host processor (notillustrated in FIG. 22) over the interconnect 2202. In an embodiment,the I/O unit 2206 communicates with the host processor directly via theinterconnect 2202 or through one or more intermediate devices such as amemory bridge. In an embodiment, the I/O unit 2206 may communicate withone or more other processors, such as one or more of the PPUs 2200 viathe interconnect 2202. In an embodiment, the I/O unit 2206 implements aPeripheral Component Interconnect Express (“PCIe”) interface forcommunications over a PCIe bus. In an embodiment, the I/O unit 2206implements interfaces for communicating with external devices.

In an embodiment, the I/O unit 2206 decodes packets received via theinterconnect 2202. In an embodiment, at least some packets representcommands configured to cause the PPU 2200 to perform various operations.In an embodiment, the I/O unit 2206 transmits the decoded commands tovarious other units of the PPU 2200 as specified by the commands. In anembodiment, commands are transmitted to the front-end unit 2210 and/ortransmitted to the hub 2216 or other units of the PPU 2200 such as oneor more copy engines, a video encoder, a video decoder, a powermanagement unit, etc. (not explicitly illustrated in FIG. 22). In anembodiment, the I/O unit 2206 is configured to route communicationsbetween and among the various logical units of the PPU 2200.

In an embodiment, a program executed by the host processor encodes acommand stream in a buffer that provides workloads to the PPU 2200 forprocessing. In an embodiment, a workload comprises instructions and datato be processed by those instructions. In an embodiment, the buffer is aregion in a memory that is accessible (e.g., read/write) by both thehost processor and the PPU 2200—the host interface unit may beconfigured to access the buffer in a system memory connected to theinterconnect 2202 via memory requests transmitted over the interconnect2202 by the I/O unit 2206. In an embodiment, the host processor writesthe command stream to the buffer and then transmits a pointer to thestart of the command stream to the PPU 2200 such that the front-end unit2210 receives pointers to one or more command streams and manages theone or more streams, reading commands from the streams and forwardingcommands to the various units of the PPU 2200.

In an embodiment, the front-end unit 2210 is coupled to a scheduler unit2212 that configures the various GPCs 2218 to process tasks defined bythe one or more streams. In an embodiment, the scheduler unit 2212 isconfigured to track state information related to the various tasksmanaged by the scheduler unit 2212 where the state information mayindicate which GPC 2218 a task is assigned to, whether the task isactive or inactive, a priority level associated with the task, and soforth. In an embodiment, the scheduler unit 2212 manages the executionof a plurality of tasks on the one or more GPCs 2218.

In an embodiment, the scheduler unit 2212 is coupled to a workdistribution unit 2214 that is configured to dispatch tasks forexecution on the GPCs 2218. In an embodiment, the work distribution unit2214 tracks a number of scheduled tasks received from the scheduler unit2212 and the work distribution unit 2214 manages a pending task pool andan active task pool for each of the GPCs 2218. In an embodiment, thepending task pool comprises a number of slots (e.g., 32 slots) thatcontain tasks assigned to be processed by a particular GPC 2218; theactive task pool may comprise a number of slots (e.g., 4 slots) fortasks that are actively being processed by the GPCs 2218 such that as aGPC 2218 completes the execution of a task, that task is evicted fromthe active task pool for the GPC 2218 and one of the other tasks fromthe pending task pool is selected and scheduled for execution on the GPC2218. In an embodiment, if an active task is idle on the GPC 2218, suchas while waiting for a data dependency to be resolved, then the activetask is evicted from the GPC 2218 and returned to the pending task poolwhile another task in the pending task pool is selected and scheduledfor execution on the GPC 2218.

In an embodiment, the work distribution unit 2214 communicates with theone or more GPCs 2218 via XBar 2220. In an embodiment, the XBar 2220 isan interconnect network that couples many of the units of the PPU 2200to other units of the PPU 2200 and can be configured to couple the workdistribution unit 2214 to a particular GPC 2218. Although not shownexplicitly, one or more other units of the PPU 2200 may also beconnected to the XBar 2220 via the hub 2216.

The tasks are managed by the scheduler unit 2212 and dispatched to a GPC2218 by the work distribution unit 2214. The GPC 2218 is configured toprocess the task and generate results. The results may be consumed byother tasks within the GPC 2218, routed to a different GPC 2218 via theXBar 2220, or stored in the memory 2204. The results can be written tothe memory 2204 via the partition units 2222, which implement a memoryinterface for reading and writing data to/from the memory 2204. Theresults can be transmitted to another PPU or CPU via the high-speed GPUinterconnect 2208. In an embodiment, the PPU 2200 includes a number U ofpartition units 2222 that is equal to the number of separate anddistinct memory devices 2204 coupled to the PPU 2200. A partition unit2222 will be described in more detail below.

In an embodiment, a host processor executes a driver kernel thatimplements an application programming interface (“API”) that enables oneor more applications executing on the host processor to scheduleoperations for execution on the PPU 2200. In an embodiment, multiplecompute applications are simultaneously executed by the PPU 2200 and thePPU 2200 provides isolation, quality of service (“QoS”), and independentaddress spaces for the multiple compute applications. In an embodiment,an application generates instructions (e.g., in the form of API calls)that cause the driver kernel to generate one or more tasks for executionby the PPU 2200, and the driver kernel outputs tasks to one or morestreams being processed by the PPU 2200. In an embodiment, each taskcomprises one or more groups of related threads, which may be referredto as a warp. In an embodiment, a warp comprises a plurality of relatedthreads (e.g., 32 threads) that can be executed in parallel. In anembodiment, cooperating threads can refer to a plurality of threadsincluding instructions to perform the task and that exchange datathrough shared memory. Threads and cooperating threads are described inmore detail, in accordance with one embodiment below.

FIG. 23 illustrates a GPC 2300 such as the GPC illustrated of the PPU2200 of FIG. 22, in accordance with one embodiment. In an embodiment,each GPC 2300 includes a number of hardware units for processing tasksand each GPC 2300 includes a pipeline manager 2302, a pre-rasteroperations unit (“PROP”) 2304, a raster engine 2308, a work distributioncrossbar (“WDX”) 2316, a memory management unit (“MMU”) 2318, one ormore Data Processing Clusters (“DPCs”) 2306, and any suitablecombination of parts. It will be appreciated that the GPC 2300 of FIG.23 may include other hardware units in lieu of or in addition to theunits shown below.

In an embodiment, the operation of the GPC 2300 is controlled by thepipeline manager 2302. The pipeline manager 2302 manages theconfiguration of the one or more DPCs 2306 for processing tasksallocated to the GPC 2300. In an embodiment, the pipeline manager 2302configures at least one of the one or more DPCs 2306 to implement atleast a portion of a graphics rendering pipeline. In an embodiment, aDPC 2306 is configured to execute a vertex shader program on theprogrammable streaming multiprocessor (“SM”) 2314. The pipeline manager2302 is configured to route packets received from a work distribution tothe appropriate logical units within the GPC 2300, in an embodiment, andsome packets may be routed to fixed function hardware units in the PROP2304 and/or raster engine 2308 while other packets may be routed to theDPCs 2306 for processing by the primitive engine 2312 or the SM 2314. Inan embodiment, the pipeline manager 2302 configures at least one of theone or more DPCs 2306 to implement a neural network model and/or acomputing pipeline.

The PROP unit 2304 is configured, in an embodiment, to route datagenerated by the raster engine 2308 and the DPCs 2306 to a RasterOperations (“ROP”) unit in the memory partition unit, described in moredetail above. In an embodiment, the PROP unit 2304 is configured toperform optimizations for color blending, organize pixel data, performaddress translations, and more. The raster engine 2308 includes a numberof fixed function hardware units configured to perform various rasteroperations, in an embodiment, and the raster engine 2308 includes asetup engine, a coarse raster engine, a culling engine, a clippingengine, a fine raster engine, a tile coalescing engine, and any suitablecombination thereof. The setup engine, in an embodiment, receivestransformed vertices and generates plane equations associated with thegeometric primitive defined by the vertices; the plane equations aretransmitted to the coarse raster engine to generate coverage information(e.g., an x, y coverage mask for a tile) for the primitive; the outputof the coarse raster engine is transmitted to the culling engine wherefragments associated with the primitive that fail a z-test are culled,and transmitted to a clipping engine where fragments lying outside aviewing frustum are clipped. In an embodiment, the fragments thatsurvive clipping and culling are passed to the fine raster engine togenerate attributes for the pixel fragments based on the plane equationsgenerated by the setup engine. In an embodiment, the output of theraster engine 2308 comprises fragments to be processed by any suitableentity such as by a fragment shader implemented within a DPC 2306.

In an embodiment, each DPC 2306 included in the GPC 2300 comprises anM-Pipe Controller (“MPC”) 2310; a primitive engine 2312; one or more SMs2314; and any suitable combination thereof. In an embodiment, the MPC2310 controls the operation of the DPC 2306, routing packets receivedfrom the pipeline manager 2302 to the appropriate units in the DPC 2306.In an embodiment, packets associated with a vertex are routed to theprimitive engine 2312, which is configured to fetch vertex attributesassociated with the vertex from memory; in contrast, packets associatedwith a shader program may be transmitted to the SM 2314.

In an embodiment, the SM 2314 comprises a programmable streamingprocessor that is configured to process tasks represented by a number ofthreads. In an embodiment, the SM 2314 is multi-threaded and configuredto execute a plurality of threads (e.g., 32 threads) from a particulargroup of threads concurrently and implements a SIMD (Single-Instruction,Multiple-Data) architecture where each thread in a group of threads(e.g., a warp) is configured to process a different set of data based onthe same set of instructions. In an embodiment, all threads in the groupof threads execute the same instructions. In an embodiment, the SM 2314implements a SIMT (Single-Instruction, Multiple Thread) architecturewherein each thread in a group of threads is configured to process adifferent set of data based on the same set of instructions, but whereindividual threads in the group of threads are allowed to diverge duringexecution. In an embodiment, a program counter, call stack, andexecution state is maintained for each warp, enabling concurrencybetween warps and serial execution within warps when threads within thewarp diverge. In another embodiment, a program counter, call stack, andexecution state is maintained for each individual thread, enabling equalconcurrency between all threads, within and between warps. In anembodiment, execution state is maintained for each individual thread,and threads executing the same instructions may be converged andexecuted in parallel for better efficiency. In an embodiment, the SM2314 is described in more detail below.

In an embodiment, the MMU 2318 provides an interface between the GPC2300 and the memory partition unit and the MMU 2318 provides translationof virtual addresses into physical addresses, memory protection, andarbitration of memory requests. In an embodiment, the MMU 2318 providesone or more translation lookaside buffers (“TLBs”) for performingtranslation of virtual addresses into physical addresses in memory.

FIG. 24 illustrates a memory partition unit of a PPU, in accordance withone embodiment. In an embodiment, the memory partition unit 2400includes a Raster Operations (“ROP”) unit 2402; a level two (“L2”) cache2404; a memory interface 2406; and any suitable combination thereof. Thememory interface 2406 is coupled to the memory. Memory interface 2406may implement 32, 64, 128, 1024-bit data buses, or the like, forhigh-speed data transfer. In an embodiment, the PPU incorporates Umemory interfaces 2406, one memory interface 2406 per pair of partitionunits 2400, where each pair of partition units 2400 is connected to acorresponding memory device. For example, PPU may be connected to up toY memory devices, such as high bandwidth memory stacks or graphicsdouble-data-rate, version 5, synchronous dynamic random access memory(“GDDR5 SDRAM”).

In an embodiment, the memory interface 2406 implements an HBM2 memoryinterface and Y equals half U. In an embodiment, the HBM2 memory stacksare located on the same physical package as the PPU, providingsubstantial power and area savings compared with conventional GDDR5SDRAM systems. In an embodiment, each HBM2 stack includes four memorydies and Y equals 4, with HBM2 stack including two 128-bit channels perdie for a total of 8 channels and a data bus width of 1024 bits.

In an embodiment, the memory supports Single-Error CorrectingDouble-Error Detecting (“SECDED”) Error Correction Code (“ECC”) toprotect data. ECC provides higher reliability for compute applicationsthat are sensitive to data corruption. Reliability is especiallyimportant in large-scale cluster computing environments where PPUsprocess very large datasets and/or run applications for extendedperiods.

In an embodiment, the PPU implements a multi-level memory hierarchy. Inan embodiment, the memory partition unit 2400 supports a unified memoryto provide a single unified virtual address space for CPU and PPUmemory, enabling data sharing between virtual memory systems. In anembodiment the frequency of accesses by a PPU to memory located on otherprocessors is traced to ensure that memory pages are moved to thephysical memory of the PPU that is accessing the pages more frequently.In an embodiment, the high-speed GPU interconnect 2618 supports addresstranslation services allowing the PPU to directly access a CPU's pagetables and provides full access to CPU memory by the PPU.

In an embodiment, copy engines transfer data between multiple PPUs orbetween PPUs and CPUs. In an embodiment, the copy engines can generatepage faults for addresses that are not mapped into the page tables, andthe memory partition unit 2400 then services the page faults, mappingthe addresses into the page table, after which the copy engine performsthe transfer. In an embodiment, memory is pinned (i.e., non-pageable)for multiple copy engine operations between multiple processors,substantially reducing the available memory. In an embodiment, withhardware page faulting, addresses can be passed to the copy engineswithout regards as to whether the memory pages are resident, and thecopy process is transparent.

Data from the memory above or other system memory is fetched by thememory partition unit 2400 and stored in the L2 cache 2404, which islocated on-chip and is shared between the various GPCs, in accordancewith one embodiment. Each memory partition unit 2400, in an embodiment,includes at least a portion of the L2 cache 2404 associated with acorresponding memory device. In an embodiment, lower level caches areimplemented in various units within the GPCs. In an embodiment, each ofthe SMs may implement a level one (“L1”) cache wherein the L1 cache isprivate memory that is dedicated to a particular SM and data from the L2cache 2404 is fetched and stored in each of the L1 caches for processingin the functional units of the SMs. In an embodiment, the L2 cache 2404is coupled to the memory interface 2406 and the XBar.

The ROP unit 2402 performs graphics raster operations related to pixelcolor, such as color compression, pixel blending, and more, in anembodiment. The ROP unit 2402, in an embodiment, implements depthtesting in conjunction with the raster engine 2408, receiving a depthfor a sample location associated with a pixel fragment from the cullingengine of the raster engine. In an embodiment, the depth is testedagainst a corresponding depth in a depth buffer for a sample locationassociated with the fragment. In an embodiment, if the fragment passesthe depth test for the sample location, then the ROP unit 2402 updatesthe depth buffer and transmits a result of the depth test to the rasterengine. It will be appreciated that the number of partition units 2400may be different than the number of GPCs and, therefore, each ROP unit2402 can, in an embodiment, be coupled to each of the GPCs. In anembodiment, the ROP unit 2402 tracks packets received from the differentGPCs and determines which result generated by the ROP unit 2402 isrouted to through the Xbar.

FIG. 25 illustrates a streaming multi-processor such as the streamingmulti-processor above, in accordance with one embodiment. In anembodiment, the SM 2500 includes: an instruction cache 2502; one or morescheduler units 2504; a register file 2508; one or more processing cores2510; one or more special function units (“SFUs”) 2512; one or moreload/store units (“LSUs”) 2514; an interconnect network 2516; a sharedmemory/L1 cache 2518; and any suitable combination thereof. In anembodiment, the work distribution unit dispatches tasks for execution onthe GPCs of the PPU and each task is allocated to a particular DPCwithin a GPC and, if the task is associated with a shader program, thetask is allocated to an SM 2500. In an embodiment, the scheduler unit2504 receives the tasks from the work distribution unit and managesinstruction scheduling for one or more thread blocks assigned to the SM2500. In an embodiment, the scheduler unit 2504 schedules thread blocksfor execution as warps of parallel threads, wherein each thread block isallocated at least one warp. In an embodiment, each warp executesthreads. In an embodiment, the scheduler unit 2504 manages a pluralityof different thread blocks, allocating the warps to the different threadblocks and then dispatching instructions from the plurality of differentcooperative groups to the various functional units (e.g., cores 2510,SFUs 2512, and LSUs 2514) during each clock cycle.

Cooperative Groups may refer to a programming model for organizinggroups of communicating threads that allows developers to express thegranularity at which threads are communicating, enabling the expressionof richer, more efficient parallel decompositions. In an embodiment,cooperative launch APIs support synchronization among thread blocks forthe execution of parallel algorithms. In an embodiment, applications ofconventional programming models provide a single, simple construct forsynchronizing cooperating threads: a barrier across all threads of athread block (e.g., the syncthreads( ) function). However, programmerswould often like to define groups of threads at smaller than threadblock granularities and synchronize within the defined groups to enablegreater performance, design flexibility, and software reuse in the formof collective group-wide function interfaces. Cooperative Groups enableprogrammers to define groups of threads explicitly at sub-block (i.e.,as small as a single thread) and multi-block granularities, and toperform collective operations such as synchronization on the threads ina cooperative group. The programming model supports clean compositionacross software boundaries so that libraries and utility functions cansynchronize safely within their local context without having to makeassumptions about convergence. Cooperative Groups primitives enable newpatterns of cooperative parallelism, including producer-consumerparallelism, opportunistic parallelism, and global synchronizationacross an entire grid of thread blocks.

In an embodiment, a dispatch unit 2506 is configured to transmitinstructions to one or more of the functional units, and the schedulerunit 2504 includes two dispatch units 2506 that enable two differentinstructions from the same warp to be dispatched during each clockcycle. In an embodiment, each scheduler unit 2504 includes a singledispatch unit 2506 or additional dispatch units 2506.

Each SM 2500, in an embodiment, includes a register file 2508 thatprovides a set of registers for the functional units of the SM 2500. Inan embodiment, the register file 2508 is divided between each of thefunctional units such that each functional unit is allocated a dedicatedportion of the register file 2508. In an embodiment, the register file2508 is divided between the different warps being executed by the SM2500 and the register file 2508 provides temporary storage for operandsconnected to the data paths of the functional units. In an embodiment,each SM 2500 comprises a plurality of L processing cores 2510. In anembodiment, the SM 2500 includes a large number (e.g., 128 or more) ofdistinct processing cores 2510. Each core 2510, in an embodiment,includes a fully pipelined, single-precision, double-precision, and/ormixed precision processing unit that includes a floating pointarithmetic logic unit and an integer arithmetic logic unit. In anembodiment, the floating point arithmetic logic units implement the IEEE754-2008 standard for floating point arithmetic. In an embodiment, thecores 2510 include 64 single-precision (32-bit) floating point cores, 64integer cores, 32 double-precision (64-bit) floating point cores, and 8tensor cores.

Tensor cores are configured to perform matrix operations in accordancewith an embodiment. In an embodiment, one or more tensor cores areincluded in the cores 2510. In an embodiment, the tensor cores areconfigured to perform deep learning matrix arithmetic, such asconvolution operations for neural network training and inferencing. Inan embodiment, each tensor core operates on a 4×4 matrix and performs amatrix multiply and accumulate operation D=A×B+C, where A, B, C, and Dare 4×4 matrices.

In an embodiment, the matrix multiply inputs A and B are 16-bit floatingpoint matrices and the accumulation matrices C and D are 16-bit floatingpoint or 32-bit floating point matrices. In an embodiment, the tensorcores operate on 16-bit floating point input data with 32-bit floatingpoint accumulation. In an embodiment, the 16-bit floating point multiplyrequires 64 operations and results in a full precision product that isthen accumulated using 32-bit floating point addition with the otherintermediate products for a 4×4×4 matrix multiply. Tensor cores are usedto perform much larger two-dimensional or higher dimensional matrixoperations, built up from these smaller elements, in an embodiment. Inan embodiment, an API, such as CUDA 9 C++ API, exposes specializedmatrix load, matrix multiply and accumulate, and matrix store operationsto efficiently use tensor cores from a CUDA-C++ program. In anembodiment, at the CUDA level, the warp-level interface assumes 16×16size matrices spanning all 32 threads of the warp.

In an embodiment, each SM 2500 comprises M SFUs 2512 that performspecial functions (e.g., attribute evaluation, reciprocal square root,and the like). In an embodiment, the SFUs 2512 include a tree traversalunit configured to traverse a hierarchical tree data structure. In anembodiment, the SFUs 2512 include a texture unit configured to performtexture map filtering operations. In an embodiment, the texture unitsare configured to load texture maps (e.g., a 2D array of texels) fromthe memory and sample the texture maps to produce sampled texture valuesfor use in shader programs executed by the SM 2500. In an embodiment,the texture maps are stored in the shared memory/L1 cache. The textureunits implement texture operations such as filtering operations usingmip-maps (e.g., texture maps of varying levels of detail), in accordancewith one embodiment. In an embodiment, each SM 2500 includes two textureunits.

Each SM 2500 comprises N LSUs that implement load and store operationsbetween the shared memory/L1 cache and the register file 2508, in anembodiment. Each SM 2500 includes an interconnect network 2516 thatconnects each of the functional units to the register file 2508 and theLSU 2514 to the register file 2508, shared memory/L1 cache 2518 in anembodiment. In an embodiment, the interconnect network 2516 is acrossbar that can be configured to connect any of the functional unitsto any of the registers in the register file 2508 and connect the LSUs2514 to the register file and memory locations in shared memory/L1 cache2518.

The shared memory/L1 cache 2518 is an array of on-chip memory thatallows for data storage and communication between the SM 2500 and theprimitive engine and between threads in the SM 2500 in an embodiment. Inan embodiment, the shared memory/L1 cache 2518 comprises 128 KB ofstorage capacity and is in the path from the SM 2500 to the partitionunit. The shared memory/L1 cache 2518, in an embodiment, is used tocache reads and writes. One or more of the shared memory/L1 cache 2518,L2 cache, and memory are backing stores.

Combining data cache and shared memory functionality into a singlememory block provides improved performance for both types of memoryaccesses, in an embodiment. The capacity, in an embodiment, is used oris usable as a cache by programs that do not use shared memory, such asif shared memory is configured to use half of the capacity, texture andload/store operations can use the remaining capacity. Integration withinthe shared memory/L1 cache 2518 enables the shared memory/L1 cache 2518to function as a high-throughput conduit for streaming data whilesimultaneously providing high-bandwidth and low-latency access tofrequently reused data, in accordance with an embodiment. Whenconfigured for general purpose parallel computation, a simplerconfiguration can be used compared with graphics processing. In anembodiment, fixed function graphics processing units are bypassed,creating a much simpler programming model. In the general purposeparallel computation configuration, the work distribution unit assignsand distributes blocks of threads directly to the DPCs, in anembodiment. The threads in a block execute the same program, using aunique thread ID in the calculation to ensure each thread generatesunique results, using the SM 2500 to execute the program and performcalculations, shared memory/L1 cache 2518 to communicate betweenthreads, and the LSU 2514 to read and write global memory through theshared memory/L1 cache 2518 and the memory partition unit, in accordancewith one embodiment. In an embodiment, when configured for generalpurpose parallel computation, the SM 2500 writes commands that thescheduler unit can use to launch new work on the DPCs.

In an embodiment, the PPU is included in or coupled to a desktopcomputer, a laptop computer, a tablet computer, servers, supercomputers,a smart-phone (e.g., a wireless, hand-held device), personal digitalassistant (“PDA”), a digital camera, a vehicle, a head mounted display,a hand-held electronic device, and more. In an embodiment, the PPU isembodied on a single semiconductor substrate. In an embodiment, the PPUis included in a system-on-a-chip (“SoC”) along with one or more otherdevices such as additional PPUs, the memory, a reduced instruction setcomputer (“RISC”) CPU, a memory management unit (“MMU”), adigital-to-analog converter (“DAC”), and the like.

In an embodiment, the PPU may be included on a graphics card thatincludes one or more memory devices. The graphics card may be configuredto interface with a PCIe slot on a motherboard of a desktop computer. Inyet another embodiment, the PPU may be an integrated graphics processingunit (“iGPU”) included in the chipset of the motherboard.

FIG. 26 illustrates a computer system 2600 in which the variousarchitecture and/or functionality can be implemented, in accordance withone embodiment. The computer system 2600, in an embodiment, isconfigured to implement various processes and methods describedthroughout this disclosure.

In an embodiment, the computer system 2600 comprises at least onecentral processing unit 2602 that is connected to a communication bus2610 implemented using any suitable protocol, such as PCI (PeripheralComponent Interconnect), PCI-Express, AGP (Accelerated Graphics Port),HyperTransport, or any other bus or point-to-point communicationprotocol(s). In an embodiment, the computer system 2600 includes a mainmemory 2604 and control logic (e.g., implemented as hardware, software,or a combination thereof) and data are stored in the main memory 2604which may take the form of random access memory (“RAM”). In anembodiment, a network interface subsystem 2622 provides an interface toother computing devices and networks for receiving data from andtransmitting data to other systems from the computer system 2600.

The computer system 2600, in an embodiment, includes input devices 2608,the parallel processing system 2612, and display devices 2606 which canbe implemented using a conventional CRT (cathode ray tube), LCD (liquidcrystal display), LED (light emitting diode), plasma display, or othersuitable display technologies. In an embodiment, user input is receivedfrom input devices 2608 such as keyboard, mouse, touchpad, microphone,and more. In an embodiment, each of the foregoing modules can besituated on a single semiconductor platform to form a processing system.

In the present description, a single semiconductor platform may refer toa sole unitary semiconductor-based integrated circuit or chip. It shouldbe noted that the term single semiconductor platform may also refer tomulti-chip modules with increased connectivity which simulate on-chipoperation, and make substantial improvements over utilizing aconventional central processing unit (“CPU”) and bus implementation. Ofcourse, the various modules may also be situated separately or invarious combinations of semiconductor platforms per the desires of theuser.

In an embodiment, computer programs in the form of machine-readableexecutable code or computer control logic algorithms are stored in themain memory 2604 and/or secondary storage. Computer programs, ifexecuted by one or more processors, enable the computer system 2600 toperform various functions in accordance with one embodiment. The memory2604, the storage, and/or any other storage are possible examples ofcomputer-readable media. Secondary storage may refer to any suitablestorage device or system such as a hard disk drive and/or a removablestorage drive, representing a floppy disk drive, a magnetic tape drive,a compact disk drive, digital versatile disk (“DVD”) drive, recordingdevice, universal serial bus (“USB”) flash memory.

In an embodiment, the architecture and/or functionality of the variousprevious figures are implemented in the context of the central processor2602; parallel processing system 2612; an integrated circuit capable ofat least a portion of the capabilities of both the central processor2602; the parallel processing system 2612; a chipset (e.g., a group ofintegrated circuits designed to work and sold as a unit for performingrelated functions, etc.); and any suitable combination of integratedcircuit.

In an embodiment, the architecture and/or functionality of the variousprevious figures is implemented in the context of a general computersystem, a circuit board system, a game console system dedicated forentertainment purposes, an application-specific system, and more. In anembodiment, the computer system 2600 may take the form of a desktopcomputer, a laptop computer, a tablet computer, servers, supercomputers,a smart-phone (e.g., a wireless, hand-held device), personal digitalassistant (“PDA”), a digital camera, a vehicle, a head-mounted display,a hand-held electronic device, a mobile phone device, a television,workstation, game consoles, embedded system, and/or any other type oflogic.

In an embodiment, a parallel processing system 2612 includes a pluralityof PPUs 2614 and associated memories 2616. In an embodiment, the PPUsare connected to a host processor or other peripheral devices via aninterconnect 2618 and a switch 2620 or multiplexer. In an embodiment,the parallel processing system 2612 distributes computational tasksacross the PPUs 2614 which can be parallelizable—for example, as part ofthe distribution of computational tasks across multiple GPU threadblocks. In an embodiment, memory is shared and accessible (e.g., forread and/or write access) across some or all of the PPUs 2614, althoughsuch shared memory may incur performance penalties relative to the useof local memory and registers resident to a PPU. In an embodiment, theoperation of the PPUs 2614 is synchronized through the use of a commandsuch as _syncthreads( ) which requires all threads in a block (e.g.,executed across multiple PPUs 2614) to reach a certain point ofexecution of code before proceeding.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed but, on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including but not limited to,”) unless otherwise noted. The term“connected,” when unmodified and referring to physical connections, isto be construed as partly or wholly contained within, attached to, orjoined together, even if there is something intervening. Recitation ofranges of values herein are merely intended to serve as a shorthandmethod of referring individually to each separate value falling withinthe range, unless otherwise indicated herein and each separate value isincorporated into the specification as if it were individually recitedherein. The use of the term “set” (e.g., “a set of items”) or “subset”unless otherwise noted or contradicted by context, is to be construed asa nonempty collection comprising one or more members. Further, unlessotherwise noted or contradicted by context, the term “subset” of acorresponding set does not necessarily denote a proper subset of thecorresponding set, but the subset and the corresponding set may beequal.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present. In addition, unless otherwise noted or contradicted bycontext, the term “plurality” indicates a state of being plural (e.g.,“a plurality of items” indicates multiple items). The number of items ina plurality is at least two, but can be more when so indicated eitherexplicitly or by context. Further, unless stated otherwise or otherwiseclear from context, the phrase “based on” means “based at least in parton” and not “based solely on.”

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. In an embodiment, a process such as thoseprocesses described herein (or variations and/or combinations thereof)is performed under the control of one or more computer systemsconfigured with executable instructions and is implemented as code(e.g., executable instructions, one or more computer programs or one ormore applications) executing collectively on one or more processors, byhardware or combinations thereof. In an embodiment, the code is storedon a computer-readable storage medium, for example, in the form of acomputer program comprising a plurality of instructions executable byone or more processors. In an embodiment, a computer-readable storagemedium is a non-transitory computer-readable storage medium thatexcludes transitory signals (e.g., a propagating transient electric orelectromagnetic transmission) but includes non-transitory data storagecircuitry (e.g., buffers, cache, and queues) within transceivers oftransitory signals. In an embodiment, code (e.g., executable code orsource code) is stored on a set of one or more non-transitorycomputer-readable storage media having stored thereon executableinstructions (or other memory to store executable instructions) that,when executed (i.e., as a result of being executed) by one or moreprocessors of a computer system, cause the computer system to performoperations described herein. The set of non-transitory computer-readablestorage media, in an embodiment, comprises multiple non-transitorycomputer-readable storage media and one or more of individualnon-transitory storage media of the multiple non-transitorycomputer-readable storage media lack all of the code while the multiplenon-transitory computer-readable storage media collectively store all ofthe code. In an embodiment, the executable instructions are executedsuch that different instructions are executed by differentprocessors—for example, a non-transitory computer-readable storagemedium store instructions and a main CPU execute some of theinstructions while a graphics processor unit executes otherinstructions. In an embodiment, different components of a computersystem have separate processors and different processors executedifferent subsets of the instructions.

Accordingly, in an embodiment, computer systems are configured toimplement one or more services that singly or collectively performoperations of processes described herein and such computer systems areconfigured with applicable hardware and/or software that enable theperformance of the operations. Further, a computer system that implementan embodiment of the present disclosure is a single device and, inanother embodiment, is a distributed computer system comprising multipledevices that operate differently such that the distributed computersystem performs the operations described herein and such that a singledevice does not perform all operations.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

Embodiments of this disclosure are described herein, including the bestmode known to the inventors for carrying out the invention. Variationsof those embodiments may become apparent to those of ordinary skill inthe art upon reading the foregoing description. The inventors expectskilled artisans to employ such variations as appropriate and theinventors intend for embodiments of the present disclosure to bepracticed otherwise than as specifically described herein. Accordingly,the scope of the present disclosure includes all modifications andequivalents of the subject matter recited in the claims appended heretoas permitted by applicable law. Moreover, any combination of theabove-described elements in all possible variations thereof isencompassed by the scope of the present disclosure unless otherwiseindicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

In the description and claims, the terms “coupled” and “connected,”along with their derivatives, may be used. It should be understood thatthese terms may be not intended as synonyms for each other. Rather, inparticular examples, “connected” or “coupled” may be used to indicatethat two or more elements are in direct or indirect physical orelectrical contact with each other. “Coupled” may also mean that two ormore elements are not in direct contact with each other, but yet stillco-operate or interact with each other.

Unless specifically stated otherwise, it may be appreciated thatthroughout the specification terms such as “processing,” “computing,”“calculating,” “determining,” or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulate and/or transform data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

In a similar manner, the term “processor” may refer to any device orportion of a device that processes electronic data from registers and/ormemory and transform that electronic data into other electronic datathat may be stored in registers and/or memory. As non-limiting examples,“processor” may be a Central Processing Unit (CPU) or a GraphicsProcessing Unit (GPU). A “computing platform” may comprise one or moreprocessors. As used herein, “software” processes may include, forexample, software and/or hardware entities that perform work over time,such as tasks, threads, and intelligent agents. Also, each process mayrefer to multiple processes, for carrying out instructions in sequenceor in parallel, continuously or intermittently. The terms “system” and“method” are used herein interchangeably insofar as the system mayembody one or more methods and the methods may be considered a system.

In the present document, references may be made to obtaining, acquiring,receiving, or inputting analog or digital data into a subsystem,computer system, or computer-implemented machine. The process ofobtaining, acquiring, receiving, or inputting analog and digital datacan be accomplished in a variety of ways such as by receiving the dataas a parameter of a function call or a call to an applicationprogramming interface. In some implementations, the process ofobtaining, acquiring, receiving, or inputting analog or digital data canbe accomplished by transferring the data via a serial or parallelinterface. In another implementation, the process of obtaining,acquiring, receiving, or inputting analog or digital data can beaccomplished by transferring the data via a computer network from theproviding entity to the acquiring entity. References may also be made toproviding, outputting, transmitting, sending, or presenting analog ordigital data. In various examples, the process of providing, outputting,transmitting, sending, or presenting analog or digital data can beaccomplished by transferring the data as an input or output parameter ofa function call, a parameter of an application programming interface orinterprocess communication mechanism.

Although the discussion above sets forth example implementations of thedescribed techniques, other architectures may be used to implement thedescribed functionality, and are intended to be within the scope of thisdisclosure. Furthermore, although specific distributions ofresponsibilities are defined above for purposes of discussion, thevarious functions and responsibilities might be distributed and dividedin different ways, depending on circumstances.

Furthermore, although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as exemplary forms ofimplementing the claims.

What is claimed is:
 1. A modular augmented reality display, comprising:a set of prescription eyewear that includes a lens to be attached to aframe; an electronic display removably attached to the prescriptioneyewear, the electronic display capable of producing a first image thatis directed into the lens so that the first image is internallyreflected by a surface of the lens; and an image combiner within thelens that is positioned to reflect the first image to a wearer, andcombine the first image with a second image transmitted through the lensand the image combiner.
 2. The modular augmented reality display ofclaim 1, further comprising a support pack connected to the prescriptioneyewear, the support pack including a battery that supplies power to theelectronic display.
 3. The modular augmented reality display of claim 2,wherein the support pack includes a microcomputer and a first wirelessinterface.
 4. The modular augmented reality display of claim 2, whereindata is transmitted from the support pack to the electronic display. 5.The modular augmented reality display of claim 4, wherein serial dataand DC power are provided to the electronic display on a singleconductor.
 6. The modular augmented reality display of claim 4, whereindata is exchanged between the support pack and the electronic displayvia a second wireless interface.
 7. The modular augmented realitydisplay of claim 1, wherein the first image is directed by a beamshaping lens from the electronic display into an in-coupling prismmounted on the lens.
 8. The modular augmented reality display of claim1, wherein the electronic display is coupled to a linear servo mechanismthat moves the electronic display to change the focal distance of thefirst image.
 9. The modular augmented reality display of claim 1,wherein the electronic display is an organic light-emitting diodedisplay or a backlit light valve technology film.
 10. The modularaugmented reality display of claim 1, wherein the electronic display ismagnetically attached to the prescription eyewear.
 11. The modularaugmented reality display of claim 1, wherein the removable attachmentincludes a mechanical alignment guide that aids in positioning theelectronic display on the prescription eyewear.
 12. The modularaugmented reality display of claim 1, further comprising a removablecontrol connected to the prescription eyewear unit having a connectionto the electronic display, the control unit including a battery thatsupplies power to the electronic display.
 13. The modular augmentedreality display of claim 2, wherein the support pack includes a flexibleconnection to the electronic display that allows weight of the supportpack to be substantially unsupported by the prescription eyewear.
 14. Aset of eyeglasses, comprising: a frame to be coupled to a lens, the lensincorporating a coupling prism; and an indexing feature to aid inaligning a removable electronic display to direct augmented realityimages to the coupling prism.
 15. The eyeglasses of claim 14, wherein aninternal image combiner is positioned such that a first image injectedinto an edge of the lens and reflected internally by a surface of thelens at least once, is reflected by the image combiner and combined witha second image transmitted through a surface of the lens to the exteriorof the lens.
 16. The eyeglasses of claim 14, wherein the eyeglassesincorporate a magnetic attachment mechanism for removably attaching theelectronic display.
 17. The eyeglasses of claim 14, wherein: a displayinterface is connected to the electronic display; and when attached tothe eyeglasses, the display interface protrudes substantiallyperpendicular to the lens.
 18. The eyeglasses of claim 14, wherein: adisplay interface is connected to the electronic display; and whenattached to the eyeglasses, the display interface protrudessubstantially parallel to the lens.
 19. The eyeglasses of claim 14,wherein: the lens has a front surface and a back surface; the lens isdivided into a first lens portion and a second lens portion by an imagecombiner; the front surface of the first lens portion is coated with aneutral density filter; and the back surface of the second lens portionis coated with the neutral density filter.
 20. The eyeglasses of claim15, wherein the electronic display is substantially flat and produces afoveated image when viewed by a wearer of the eyeglasses.
 21. Theeyeglasses of claim 20, wherein: the electronic display has a uniformresolution; and the pixel density distribution of the foveated image isoptically produced at least in part by the image combiner.
 22. Anaugmented reality display, comprising: a set of eyeglasses that includea lens attached to a frame; an electronic display attached to theeyeglasses, the electronic display capable of producing a first imagethat is directed into the lens so that the first image is internallyreflected by a surface of the lens; an image combiner within the lensthat is positioned to reflect the first image to a wearer, and combinethe first image with a second image transmitted through the lens and theimage combiner; and a belt pack that includes a battery and amicrocomputer that provides display data to the electronic display. 23.The augmented reality display of claim 22, further comprising amechanical attachment that allows the electronic display to berepeatedly detached from the eyeglasses.
 24. The augmented realitydisplay of claim 23, wherein the mechanical attachment is a magneticattachment with indexing features that align the electronic display withthe lens.
 25. The augmented reality display of claim 22, wherein: thelens is adapted to an eyeglass prescription of the wearer; and a surfaceof the image combiner is adapted to the eyeglass prescription of thewearer.
 26. A set of eyeglasses, comprising: a lens to be coupled to aframe, the lens incorporating a coupling prism; and an indexing featureto aid in aligning a removable electronic display to direct augmentedreality images to the coupling prism.
 27. The eyeglasses of claim 26,wherein the lens includes a magnetic insert for attaching the electronicdisplay to the lens.